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ABSTRACT 



This report shows how lattice theory can be used to develop 
quantitative measures of selected characteristics of knowledge structure 
representations and how these measures can be used to assess individual 
person's knowledge structure representations in a classroom setting. For a 
given set of concepts, a knowledge structure can be described by the present 
or absent connections in the set of all possible pairwise connections between 
concepts. Under this description, the set of all possible knowledge structure 
representations for a given set of concepts forms the elements of a 
complemented, distributive lattice ordered by set inclusion. Measures are 
developed to assess the dissimilarity between two knowledge structure 
representations, the local complexity of a concept in a knowledge structure, 
and the global complexity of a knowledge structure. The effectiveness of 
these measures in assessing the changes in students ' knowledge structure 
representations in an introductory statistics course is examined using data 
from T. Ju (1989). (Contains 6 tables, 9 figures, and 39 references.) 
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Abstract 

This work shows how lattice theory can be used to develop quantitative measures of 
selected characteristics of knowledge structure representations, and how these 
measures can be used to assess individual persons' knowledge structure representations 
in a classroom setting. For a given set of concepts, a knowledge structure can be described 
by the present or absent connections in the set of all possible pairwise connections 
between concepts. Under this description, the set of all possible knowledge structure 
representations for a given set of concepts are the elements of a complemented, 
distributive lattice ordered by set inclusion. Measures are developed to assess the 
dissimilarity between two knowledge structure representations, the local complexity of 
a concept in a knowledge structure, and the global complexity of a knowledge structure. 

The effectiveness of these measures in assessing the changes in students' knowledge 
structure representations in an introductory statistics course is examined using data from 
Ju (1989). 

Introduction 

The ability to quantitatively assess individuals' knowledge has important 
implications for educational measurement, diagnostic assessment, and test 
development. Snow and Lohman (1989) stress the importance of knowledge 
structure assessment to the educational process: 

Students build up vast structures of particularized knowledge, both declarative and 
procedural, over their educational years... Such knowledge is often partial, 
incomplete, or incorrect in idiosyncratic ways. It is also often tied to particular 
situations. And it can be brought -into new learning in ways that distort the new 
learning, as well as in ways that allow new learning to complete or correct or the 
supplant the old. The improvement in knowledge assessment would seem to depend 
generally on diagnosis, both of prior knowledge and of knowledge in the process of being 
acquired. Such assessment should include direct attempts to assess how concepts are 
organized... (p. 304) 



Knowledge structure assessment for diagnostic purposes is seen by some as a 
corrective to the shortcomings of traditional testing practices. Surber (1984) has 
suggested that traditional testing for diagnosis of misunderstanding suffers from 
the drawbacks of (a) insensitivity to structure, (b) intrusions and distortions, and 
(c) errors of omission. Traditional tests are often insensitive to the structure of a 
subject being tested, because any given subject “is not merely a collection of lists 
of concepts. A discipline is recognized as such because of the interrelationships of 
its concepts" (Surber, 1984, p. 215). Items on a multiple choice test are often 
inadequate to test a subject's structural knowledge of a given domain. The 
alternative of short answer or essay questions may also have problems. In some 
cases, information that was not a part of the instruction may act as an intrusion 
in the response given, while in others, information may be distorted in the 
process of giving an answer. In either case, these test formats provide: 

... no general method of dealing with declarative knowledge errors in terms of scoring or 
for diagnostic purposes... [T]hese error responses do not lead to a better understanding of 
what the learner knows. In short, an essay test does not permit the systematic diagnosis 
of a learner's misunderstanding. (Surber, 1984, p. 215) 

Finally, traditional tests often fail to distinguish between a lack of 
knowledge and erroneous knowledge; in the case of the multiple choice tests, 
this may be further confounded by the results of guessing. Additionally, since 
multiple choice items are often weighted equally, there is no way to assess 
systematically the importance of the missing knowledge. The need to diagnose 
for errors of omission is essential if this is to be remedied. 

The importance of knowledge structure assessment to the instructional 
design of diagnostic tests has also been addressed. Nitko (1989) argues that 
eliciting representations of learners' knowledge structures and comparing them 
to the "canonical" knowledge structures of experts should be an integral part of 
diagnostic assessment and instructional design: 

A test designer's understanding of the meaning and structure of the knowledge a student 
brings to the instructional system is important for building diagnostic tests. Tests of 
prerequisites should focus on these aspects of the preinstructed learner. Frequently, 
students' everyday understandings of terms and phenomena are at odds with the 
experts' canonical understandings. These conflicts can interfere with instruction 
directed toward acquisition of canonical knowledge, unless students' knowledge schema 
are explicitly addressed in the course of teaching, (p. 461) 



While many writers have recognized the importance of knowledge structure 
assessment (e.g., Glaser, Lesgold, & Lajoie, 1987; Lane, 1991), it is still the case that 
"cognitive theories about knowledge structures have progressed far ahead of 
research on methods for their assessment that would be useful in education" 
(Snow & Lohman, 1989, p. 304). 

In recent years a number of methods have been developed to elicit a 
representation of a subject's structural understanding for a set of concepts. These 
techniques have used a simplified model to represent structural knowledge as a 
network of interconnected nodes. In the model's basic form, each node has 
represented a concept, the meaning of which is determined by its connections to 
other concepts (nodes) in the network. Using this model, these techniques elicit 
knowledge structure representations 1 for a set of concepts by requiring a subject 
to directly manipulate the nodes representing the concepts, and the connections 
representing the relationships among them. 

While these methods have been used successfully to elicit knowledge 
structure representations in such knowledge domains as statistics (Rogan, 1988; 
Ju, 1989), the biological sciences (Fisher, Faletti, Thornton, Patterson, Lipson, & 
Spring, 1988), physics (Jonassen, 1987; Hegarty-Hazel & Prosser, 1991), chemistry 
(Ruiz-Primo, Schultz, & Shavelson, 1997; Ruiz-Primo, Shavelson, & Schultz, 
1997), and geology (Champagne & Klopfer, 1980; Ballstaedt & Mandl, 1985), the 
analyses of these representations have often been problematic. Most studies have 
transformed the data into proximity matrices in order to use a scaling method 
such as hierarchical cluster analysis or multidimensional scaling. When these 
matrices have been derived by averaging the data across the subjects, group 
information has been obtained at the expense of losing the characteristics of the 
individual knowledge structures. Those studies that have not used proximity 
matrices (e.g.. Champagne & Klopfer, 1980) have relied on analytic techniques 
that are qualitative rather than quantitative. 



1 What will be referred to as "structural knowledge" in this paper has several different definitions 
in the literature. Some writers have associated structural knowledge exclusively with declarative 
knowledge (e.g., Snow & Lohman, 1989, p. 298), while others have proposed structural knowledge 
as an intermediate type of knowledge that mediates the translation of declarative into procedural 
knowledge (Jonassen, Beissner, & Yacci, 1993, p. 4). Structural knowledge has also be referred to as 
cognitive structure, conceptual knowledge, and semantic networks (Jonassen et al., 1993, p. 5). In any 
event, the representation of such structural knowledge is distinct from the actual underlying 
cognitive model. 



